Methods of Comparability Studies for Computerized and Paper-Based Tests

In recent years, tests have begun being administered by computer. The potential benefits of computerized testing include more efficient test administration, flexible scheduling, quicker score reporting, more accurate examinee ability estimation, and expanded content/construct coverage. Although computerized testing is an enticing option, most testing programs find it necessary to offer both computer and paper versions of a test at one time, sometimes basing similar decisions on scores derived from both testing media. In this context, a comparability study must be conducted to determine if scores from the computer and paper versions of the test are equivalent. Professional testing standards (APA, 1986; AERA, APA, & NCME, 1999) and the Standards and Assessments Peer Review Guidance (USED, 2007) stress the need to study score comparability across test administration media. The first and most critical step in any comparability study is to collect good data for making score comparisons, and three data collection designs are commonly employed.